Skip to content

fix(remote-delegation): a reply from a linked environment on another version still completes its task - #540

Merged
scuttletoby merged 5 commits into
mainfrom
fork/386-reply-read-other-version
Nov 19, 2026
Merged

scuttletoby merged 5 commits into
mainfrom
fork/386-reply-read-other-version

Conversation

@scuttletoby

Copy link
Copy Markdown
Owner

Part of #386. This is part 2. Part 1 was #412: the follow survives an interrupted delegating call.

The bug

A task delegated to a linked environment that runs an older T3 build never completes here, so its parent is never told.

Here is why. When the remote run ends, the follower reads the task's thread there with t3_thread_read to get its reply. This version decodes that answer strictly, and the whole thread is decoded, including the thread's details. Upstream PR 34895 made two fields of a linked pull request required: watching and watchStartedAt. An older build does not send them. So any thread there with a linked PR fails the read with "The linked environment's answer did not match this version. Update both environments." The follower treats that as retryable and retries forever. The task stays running here, and the result never arrives.

The fix (design choice)

The change is in apps/server/src/fork/remoteReplyRead.ts. It adds 3 seam lines to RemoteDelegation.ts, announced in #538.

  • lastReply reads through a small wrapper. It first does the full read, as before. If that fails as orchestration_error (an answer this version cannot decode), it does the same t3_thread_read again through PeerForwarding.call. This time the answer is decoded as the thread-read result without thread: just runs, items and paging.
  • A reply never needs the thread's details. Runs and timeline items keep their full schemas, so a reply is never guessed or half-decoded.
  • If the second read also fails, the first error stands, and the follower behaves exactly as today. The same holds if call refuses because linked environments are off here.

Options I did not take:

  • Make the two fields optional in the contract. Upstream already does this for other fields ("Absent from linked environments that predate it"). It is the proper general fix, and it would also fix the agent-facing t3_thread_read/t3_thread_list against older peers. But it changes lines in a pending upstream contracts file, which the fork may not grow. That is for upstream.
  • Decode everything leniently. Rejected: that could deliver a wrong or partial reply.
  • Give up after N retries. Rejected: the task would still lose its real result.

State question

  • Runs again: the read has no side effects. Completion keeps its existing guard, so a task completes once.
  • Stops half-way: a restart refollows open tasks through start(), and they read their reply the same way.
  • Old state: tasks already stuck running because of this complete on the next start that has this build. Their result is the child's last reply, which may be old by then.
  • Empty: a task with no run, or no reply, takes the same path as before.
  • Newer peer: extra fields were already ignored, so nothing changes.

Tests

apps/server/src/fork/remoteReplyRead.test.ts runs a real RemoteDelegation.layer follow against a mocked peer:

  • Current-version answer: completes with the reply in one read.
  • Older-version answer: completes with the reply, using the second read. This test fails on main (the task ends failed after the retry schedule) and passes with the fix.
  • An answer whose items cannot be decoded either, and the same with linked environments off: the task still fails as before, with the original error, not a new one.

RemoteDelegation.test.ts, the #412 tests and these all pass (51/51). vp run check:changed passes.

Upstream fit

  • No upstream issue or PR fixes this. Upstream PR 34895 added the required fields. Upstream PR 30249 (record before launch) and upstream issue 29860 touch the same delegation code but not the reply read.
  • The fix builds on upstream's own pieces: PeerForwarding.call with a tool's success schema, the OrchestratorMcpThreadReadResult schema (mapped with Struct.omit), and the existing retry schedule and followGaveUp. It adds 3 seam lines in RemoteDelegation.ts and no change to upstream lines.
  • The general fix belongs upstream: optional watching/watchStartedAt, as upstream does for other fields that older linked environments lack. If upstream does that, this module and its 3 seam lines can be dropped.

General vs project

This is general. Any two linked machines on different builds hit it. It has no setting and nothing project-specific.

Merge order

Merge #538, the list-only seam announce, first. Until then this PR's seam check fails by design, since it is judged by the list in the base. This branch contains #538's commit, so after #538 merges it shows only the fix.

Not covered here (rest of #386)

  • Agent-facing t3_thread_read / t3_thread_list against an older peer still fail. That needs the contract change above, upstream. Updating the older machine also fixes it.
  • A "completed" notice that holds only an interim progress note. The child ended its turn while background work was still running and unawaited. Its later run, woken by that work, never reports to the parent (upstream design). This is an owner decision. My recommendation is no code change: children wait with t3_background_wait before ending a turn.
  • At-least-once delivery and a reconcile of parent and child state. That is the next part.

🤖 Generated with Claude Code

scuttletoby and others added 2 commits November 19, 2026 15:21
List only. Announces the three lines the follow-up pull request adds to
RemoteDelegation.ts: an import of fork/remoteReplyRead.ts, the wrapped
forwarding, and lastReply reading through it. A task delegated to a
linked environment on another version then still delivers its reply.

Part of #386.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…version still completes its task

A task delegated to a linked environment that runs an older T3 build never
completed here. Its thread read decodes the whole thread, and a field this
version requires in the thread's details (a linked pull request's watch
state) is missing there, so every read failed as "did not match this
version" and the follower retried forever: the parent was never told.

lastReply now reads through fork/remoteReplyRead.ts. When the full read
fails as an answer this version cannot decode, it reads the same page again
without the thread's details, which a reply never needs. Runs and items keep
their own schemas, so a reply is never guessed; if that read fails too, the
first error stands as before.

Moves the three announced RemoteDelegation.ts lines to the seam's added list.

Part of #386.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (independent, Opus)

Head reviewed: 35869d0d08. Read #386, RemoteDelegation.ts (follow, followOnce, lastReply, finalReason), PeerForwarding.ts, PeerMcpClient.ts and the new module and test.

Verdict: 1 blocking finding. The fix itself is sound. CI is red because of a fork test this PR breaks; it is a one-line fix.

Findings, most severe first

1. BLOCKING: Test Server 6 fails on fork/personOnlySettings.test.ts, "finds every T3 tool".
apps/server/src/fork/remoteReplyRead.ts:22 calls Tool.make outside the toolkit modules. The guard at apps/server/src/fork/personOnlySettings.test.ts:173 expects every such file to be listed in NOT_ADVERTISED (personOnlySettings.test.ts:77-80), and only peer/RemoteDelegation.ts is listed.

  • Failure: CI gets expected [ 'fork/remoteReplyRead.ts', 'peer/RemoteDelegation.ts' ] to deeply equal [ 'peer/RemoteDelegation.ts' ], so the PR cannot merge. vp run check:changed did not catch it because that test does not sit beside a changed file.
  • Fix: add "fork/remoteReplyRead.ts": "a client of another environment's tool", with a one-line comment such as "Decodes the linked environment's t3_thread_read without the thread's details."

2. NIT: the error match is wider than the bug (every orchestration_error, not just the version mismatch). apps/server/src/fork/remoteReplyRead.ts:41-47.
In PeerMcpClient.ts, orchestration_error also covers "stopped answering" (transport error or timeout), any non-200 HTTP answer, a 403 because linked environments are off there, "unexpected shape", a peer refusal with that code, and link-storage errors. Each of these now sends a second t3_thread_read.

  • Correctness is not affected. The second read has the same input. Runs, items and paging keep their strict schemas. If it fails, the first error is restored (line 46), so finalReason/isFinal and the retry schedule see exactly what they saw before. If it succeeds after a transient first failure, it returns the same runs and items the full read would have returned.
  • Cost: while a peer is down or erroring, each follow attempt makes two requests and records the link outcome twice. Forwarded tool calls have no time limit of their own (PeerMcpClient.ts:163: only the handshake is bounded), so a hanging peer can take about twice as long per attempt before back-off.
  • Option: also match error.message.startsWith("The linked environment's answer did not match this version") (PeerMcpClient.ts:279). The trade-off: the test mock copies that text (remoteReplyRead.test.ts:143-147), so if upstream rewords it, the fallback would silently stop working while the tests stay green. If you narrow the match, the test should take the message from the real client. Keeping the match as it is also defensible.

3. NIT: with linked environments turned off here, an older peer's task still fails. PeerForwarding.ts:234-251.
readThread deliberately keeps working while the beta is off ("Tasks already running there keep following through readThread..."). The fallback goes through call, which checks requireEnabled first.

What I checked and found sound

  • Can a wrong or partial reply become final? No. lastReply runs only after the wait reports a terminal run (RemoteDelegation.ts:838, :852, :878), and the run filter by runId is unchanged. ReplyPage omits only thread, so each page decodes completely or not at all. A reply that spans pages, some read in full and some through the fallback, carries the same items either way. The extra thread key is ignored on decode, as the "older answer" test shows. I checked the contract history: OrchestratorMcpThreadRun and OrchestratorMcpThreadTimelineItem gained no new required fields recently. The required fields that cause the failure (watching, watchStartedAt, from upstream PR 34895) are only in the thread details, so the fallback covers the real drift.
  • Are other decode errors swallowed? No. The fallback triggers only on orchestration_error. The final codes (capability_denied, thread_not_found, run_not_found) and invalid_request pass straight through. A failed fallback restores the original error.
  • Is the same-version path unchanged? Yes. The first call is upstream's forwarding.readThread, and catchIf only acts on its failure. The shadowing const forwarding = replyForwarding narrows the type inside lastReply to readThread and to a result without thread. If upstream later reads read.thread or another forwarding method inside lastReply, the typecheck fails loudly during fork-follow instead of misbehaving.
  • State question:
    • Retried: the read has no side effects, and completion keeps its guard.
    • Parent restarts mid-read: start() follows the open tasks again, and they read their reply the same way.
    • Remote goes away: both reads fail, the first error stands, and back-off and give-up behave as before (with the doubled requests from finding 2).
    • Old stuck task completes late: a task that already gave up has a result and is not followed again. A task still running completes with the reply of its own run, filtered by runId, and the sharedThread logic is untouched. The parent may get a late notice for an old task, which is the intended at-least-once behaviour.
  • Tests, run locally:
    • The three files (remoteReplyRead, RemoteDelegation, remoteDelegationAbortSafe) pass on the PR: 51/51.
    • With only RemoteDelegation.ts reverted to main, "an older version's answer still completes the task with its reply" fails (status: 'failed'), and the other 3 cases pass. So the test fails on main and passes with the fix, as claimed.
  • Seam check: the red "No database schema changes" and "Workflow lint and upstream drift" runs predate the merge of fork seams: announce the #386 reply-read lines in RemoteDelegation.ts #538. Run locally against current main (in both the --config-ref and the HEAD^1 HEAD modes), the seam check passes. Those jobs need a branch update or a rerun.

The three RemoteDelegation.ts lines main announced (#538) move to
"added", as this change adds them; the person-only settings test lists
fork/remoteReplyRead.ts as a client of another environment's tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

scuttletoby commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner Author

Fix round 1

Item Fix Where
BLOCKING: "finds every T3 tool" fails on the new Tool.make fork/remoteReplyRead.ts listed in NOT_ADVERTISED as a client of another environment's tool, with a one-line reason apps/server/src/fork/personOnlySettings.test.ts:80-81
DIRTY after #538 Merged origin/main. #538 announced the three RemoteDelegation.ts lines; this PR adds them, so they move from "announced" to "added" (as #528 did after #533) .github/fork-seams.json:126-128
Nit: fallback only for the version-mismatch error Left as is. The error can be told apart only by its message text (PeerMcpClient.ts:279), which the test copies, and that is more fragile than the orchestration_error code. The fallback cannot widen a result: its schema only drops thread, and if it fails, the first error stands. apps/server/src/fork/remoteReplyRead.ts:41

Local runs (vp test run): personOnlySettings, remoteReplyRead, RemoteDelegation, remoteDelegationAbortSafe: 4 files, 62 tests passed.
State: only a test list and the seam list changed; runtime behaviour is unchanged.

Main moved again (#268, #541): merged once more; FORK_CHANGES.md keeps both new rows. Same 4 test files re-run: 62 passed.

scuttletoby and others added 2 commits November 19, 2026 16:48
FORK_CHANGES.md keeps both new rows (this PR's linked-task row and #268's
model policy row).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby merged commit 0757ca9 into main Nov 19, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant